Flow-Based Load Balancing

ABSTRACT

Methods and apparatuses are described for virtualizing routing of network traffic by offloading routing decisions to a controller in communication with a plurality of network devices. For load balancing applications, the controller may make up-front decisions as to both destination and route, rather than wait until traffic has been routed to a load balancing point before determining the destination.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority to U.S.application Ser. No. 17/360,983, filed Jun. 28, 2021, which is acontinuation of U.S. application Ser. No. 15/891,798, filed Feb. 8, 2018(now U.S. Pat. No. 11,050,668), which is a continuation of U.S.application Ser. No. 15/360,618, filed Nov. 23, 2016 (now U.S. Pat. No.9,929,961), which is a continuation of U.S. application Ser. No.14/306,783, filed Jun. 17, 2014, (now U.S. Pat. No. 9,548,927). Theentire contents of these applications are incorporated herein byreference in their entireties.

BACKGROUND

Commercially available hardware load balancers that act as a gateway fornetwork traffic are well-known. These load balancers typicallydistribute network traffic amongst a collection of directly-attachedservers directly by rewriting the destination address of incomingpackets to that of the desired destination server. However, such loadbalancers are generally not very good at distributing amongstgeographically disparate destination servers, at least not withoutrouting traffic in an inefficient way such as by “hair-pinning” thetraffic back outward into the wide area network. Even then, conventionalload balancing relies on a single instance of the hardware device or alogical cluster at a single physical location. There is typically nogeographically diverse failover.

While techniques such as so-called “global” load balancing (which uses adomain name system to redirect traffic) and Anycast (which advertises anIP address into routing protocols from multiple origin locations), aresometimes used to supplement the single-instance hardware loadbalancers, even systems using a combination of all three techniquesstill suffer from load balancing challenges such as inefficient routingover large geographic areas and single points of failure. Moreover,recent limited experiments in OpenFlow-based load balancing have failedto successfully address these and other problems.

SUMMARY

The following summary is for illustrative purposes only, and is notintended to limit or constrain the detailed description.

Load balancing and routing of traffic to and from a pool ofload-balanced resources may be integrated. Rather than routing trafficto a traditional load balancer and then having the load balancerdetermine where to distribute the load, both the route and thedestination may be determined together. In fact, both the route and thedestination may be determined at the first instance that a new packet isdetected in a network for which a route and destination have not alreadybeen established.

Packets may be logically arranged in flows, such as flows of datapackets between the same source address and destination address. Foreach packet that is part of a flow that the network device does not haveinstructions for, a network device receiving the packet may query acontroller to determine how to handle the packet. The controller, inresponse to the query, may determine a destination and a route to and/orfrom that destination, and the controller may respond to the networkdevice with instructions on how to forward that packet to the next hopin the route, which may also apply to future packets in the same flow.The controller may or may not also instruct other network devices in thedetermined route as to how to forward the packet. Where a network devicereceives a packet of a flow for which the network device hasinstructions, the network device may follow those instructions withoutnecessarily needing to contact the controller. As will be describedherein, the controller may be configured to use information about thesoftware application utilizing the flow and/or information about thenetwork and/or the servers such as network congestion, serverutilization, and/or other information, to make a holistic determinationof both the destination for network traffic in a flow and the route tothe destination.

Accordingly, aspects as described herein are directed to, for example,determining, based at least on a destination address indicated by a datapacket, a plurality of routes to a plurality of destinations, andselecting a first route of the plurality of routes to a firstdestination of the plurality of destinations. Instructions may be sent,such as from the controller to one of the network devices, to route thedata packet along at least a portion of the first route.

Further aspects as described herein are directed to, for example,determining a route to one of a plurality of destinations eachassociated with a destination address of a data packet, and selecting,from a plurality of network devices, a set of network devices along theroute. Instructions may be sent, such as by the controller, to the setof network devices for routing the data packet along at least a portionof the route.

Still further aspects as described herein are directed to, for example,the following. Responsive to a message from a first network deviceidentifying at least a portion of a header of a data packet, the atleast a portion of the header may be used to determine a plurality ofroute/destination pairs for a plurality of destinations. A firstroute/destination pair may be selected from the plurality ofroute/destination pairs, and instructions may be sent, such as from thecontroller, to the first network device for forwarding the data packetto a second network device in a route of the first route/destinationpair.

The summary herein is not an exhaustive listing of the novel featuresdescribed herein, and is not limiting of the claims. These and otherfeatures are described in greater detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood with regard to the followingdescription, claims, and drawings. The present disclosure is illustratedby way of example, and not limited by, the accompanying figures in whichlike numerals indicate similar elements.

FIG. 1 illustrates an example system in which various features describedherein may be implemented in accordance with one or more aspects asdescribed herein.

FIG. 2 illustrates another example system in which various featuresdescribed herein may be implemented in accordance with one or moreaspects as described herein.

FIGS. 3A-3C illustrate the system of FIG. 2 with an example ofperforming a process in accordance with one or more aspects as describedherein.

FIG. 4 illustrates the system of FIG. 2 with an example route inaccordance with one or more aspects as described herein.

FIG. 5 illustrates the system of FIG. 2 with another example route inaccordance with one or more aspects as described herein.

FIG. 6 illustrates an example computing device that can be used toimplement any of the methods, servers, client devices, network devices,and other elements discussed herein and shown in the figures, inaccordance with one or more aspects as described herein.

FIGS. 7-9 illustrate steps of an example process that may be performedin accordance with one or more aspects as described herein.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be used, and structural andfunctional modifications may be made, without departing from the scopeof the present disclosure.

FIG. 1 illustrates an example system in which various features describedherein may be implemented in accordance with one or more aspects asdescribed herein. The system of FIG. 1 includes one or more clientdevices (such as a client device 101), a load balancing/routing function102, and a plurality of servers (such as Server A, Server B, Server C,Server D, and Server E), which may be, for example, origin serversarranged as one or more pools of origin servers. While five servers areshown in the present example, there may be fewer than or more than fiveservers.

The client device 101 may be any type of computing device, such as butnot limited to a personal computer (e.g., a desktop computer, a laptopcomputer, a tablet computer, etc.), a smart phone (e.g., a cellularphone), a video gaming system, an automobile computer, a GPS device, aset top box, a television set, a server, a gateway, a modem, a router, aswitch, a controller, and/or the like. The client device 101 may becommunicatively coupled with the load balancing/routing function 102via, for example, one or more wired communication lines, one or morenetworks, and/or one or more wireless links. Where a network is used tocommunicate between the client device 101 and the load balancing/routingfunction 102, the client device 101 may be outside the network or adevice that is part of the network such as a network router or networkswitch.

The load balancing/routing function 102 may be implemented as one ormore computing devices, such as one or more servers, routers, switches,gateways, and/or other types of computing devices. While illustrated inFIG. 1 as a single block, the load balancing/routing function 102 mayactually comprise a plurality of interconnected and distributed physicaldevices and, as will be described below in greater detail, may beimplemented using the same devices that make up a network fordistributing messages between the client device 101 and Servers A-E.

While servers A-E are labeled as “servers” in the example of FIG. 1 ,servers A-E may each be any types of one or more computing devices, suchas but not limited to servers (such as origin servers), personalcomputers, routers, switches, gateways, output devices, and/or the like.Moreover, any of servers A-E may be geographically co-located with oneanother or geographically dispersed from one another, as desired. Forexample, two or more of servers A-E may be in separate states, thousandsof miles from each other, coupled to different local networks (that arein turn coupled to each other through one or more wider area networks),in different countries, etc. As will be discussed further below, theholistic integration of load balancing and routing decisions, as may beaccomplished herein, may allow for a plurality of servers to be part ofthe same load balancing pool, regardless of whether the servers in thepool are geographically (and/or topologically in the network) disparatefrom one another.

In operation, the client device 101 may send a message toward loadbalancing/routing function 102 directed to a destination associated witha pool of servers such as Servers A-E, and the load balancing/routingfunction 102 may selectively distribute the message to one (or a subsetof) the pool of servers. For example, the message may be sent as one ormore data packets, such as by a web browser executing on the clientdevice 101 requesting information from a web page at a certain virtualaddress that happens to be associated with the server pool of serversA-E. In such a case, two or more of servers A-E may each store the webpage and may each act as a web server for the requested web page. Inresponse to the message, the load balancing/routing function 102 maymake a decision as to how to distribute the message, and to which one ormore of servers A-E to distribute the message to, based on one or morefactors. The one or more factors may include, for example, currentreported utilizations or other status of one or more of the servers,predicted future utilizations or other status of one or more of theservers, congestion, errors, other monitored states of one or morecommunication paths between the client device 101 and one or more of theservers, an identity of the client device 101 and/or of a user of theclient device 101, a characteristic of the message, the time of day, theday of week, and/or a preference of the user of the client device 101and/or otherwise associated with the client device 101. In response tothe message, the one or more of servers A-E that receive the message mayreply with their own message.

FIG. 2 illustrates another example system in which various featuresdescribed herein may be implemented in accordance with one or moreaspects as described herein. In some embodiments, FIG. 2 may be a moredetailed illustration of the system of FIG. 1 . For example, the clientdevice 101 in FIG. 1 may be the same client device as any of clientdevices 201-204 in FIG. 2 , and the load balancing/routing function 102may be implemented by one or more of the elements of FIG. 2 , such as byany of elements 205-216 and 220. Moreover, servers A-E of FIG. 1 may bethe same servers as servers A-E of FIG. 2 . In other embodiments, thetwo systems of FIGS. 1 and 2 are entirely separate embodiments.

The example system of FIG. 2 includes one or more client devices (suchas client devices 201, 202, 203, and 204), a network 200 comprising oneor more network devices (such as network devices 205-216), and one ormore controllers (such as controller 220) that may be part of network200 and/or outside of network 200. Each of the client devices 201-204may be implemented as one or more computing devices, such as but notlimited to a personal computer (e.g., a desktop computer, a laptopcomputer, a tablet computer, etc.), a smart phone (e.g., a cellularphone), a video gaming system, an automobile computer, a GPS device, aset top box, a television set, a server, a gateway, a modem, a router, aswitch, a controller, and/or the like.

Each of the network devices 205-216 may also be implemented as one ormore computing devices, such as but not limited to a router, a switch, agateway, etc. In general, each of the network devices 205-216 may beconfigured to receive messages and forward those message (modified orintact) to another device. For example, the network device 205 may beconfigured to forward messages received from any of the client device201, the network device 206, and/or the network device 213 to any one ormore of the client device 201, the network device 206, and/or thenetwork device 213. One or more of the network devices (for example, thenetwork devices 205, 208, 210, 211, 214, and/or 216) may be edge routersand may provide routing and/or switching functionality between thenetwork 200 and devices outside of the network 200. One or more othersof the network devices (for example, the network devices 207, 209, 212,213, and/or 215) may be core routers within the network 200.

While the network 200 is represented in FIG. 2 as a single cloud, thenetwork 200 may be a single network or a collection of multipleconnected networks. By way of example, the network 200 may be orotherwise include the Internet and/or another IP-based network, anintranet, a telephone network, a local area network (LAN), a wide areanetwork (WAN), a wireless (e.g., cellular network), a satellite network,an Ethernet network, a wi-fi network, an optical fiber network, acoaxial cable network, and/or a hybrid fiber/coax distribution network.In some embodiments, core routers (e.g., the network device 213) may bea core router of the Internet backbone and edge routers (e.g., thenetwork device 205) may provide a client device (e.g., the client device201) with access to the Internet. Others of the network devices mayprovide connectivity between the core routers and the edge routers.Also, while the various client devices 201-204 are shown as beingcommunicatively coupled to a respective network device (such as thenetwork device 205), each client device may also be part of a localnetwork (such as an Ethernet and/or wi-fi network) that providesconnectivity between the client device and the edge router.

In the example of FIG. 2 , while the controller 220 is depicted as asingle block, from a physical standpoint the controller 220 may beimplemented as a single device (such as a single server or personalcomputer) or as a plurality of distributed devices (such as a pluralityof servers and/or personal computers). Moreover the controller 220 maybe part of the network 200 or outside of the network 200. In eithercase, the controller 220 may be communicatively coupled to one or moreof the network devices 205-216. Moreover, the controller 220 may bephysically integrated into and/or share computing resources with any oneor more of the network devices 205-216 or be a physically separatecomputing device from the network devices 205-216.

As will be described further below, the controller 220 may be configuredto provide one or more of the network devices 205-216 with instructionsfor routing packets through the network 200. In operation, when one ofthe network devices 205-216 receives a packet that is not part of anexisting flow (e.g., an existing connection), then the network devicemay send a request message (such as through the network 200) to thecontroller 220 for instructions on how to route packets for the flow. Inresponse, the controller 220 may determine one or more candidatedestinations, determine how to route such packets to the one or morecandidate destinations, choose one of the candidate destination/routepairs, and send a response message to one or more of the network devices(including the requesting network device) indicating how to route thepackets for the flow based on the chosen candidate destination/routepair. Thus, rather than network devices in a routing path making theirown routing decisions, those decisions may be offloaded to thecontroller 220 that is not in-line with the routing path and that mayhave knowledge of relevant information about the network that may not bedirectly available to the network devices. Compared with traditionalload balancers that are in-line with the routing path and are typicallyone or two hops away from the origin server (the ultimate destination),the controller 220 may be located anywhere inside or outside of thenetwork 200, close to the ultimate destination or far from thedestination, and in-line with the routing path or outside of the routingpath, as desired. In fact, the route and the ultimate destination may becompletely determined even before the packet is routed by the networkdevice. To accomplish this, a receiving network device may inform thecontroller 220 about incoming packets that are not part of existingflows (to the knowledge of the network device), and the controller 220may instruct that network device and any other network devices along adetermined routing path what to do with the packets. In someembodiments, the routing request/response function may be performedusing OpenFlow or another type of communications protocol suitable forsuch a purpose. Thus, in such embodiments, the controller 220 may beconsidered an OpenFlow controller. In making routing decisions, thecontroller 220 may have knowledge of information about the network suchas network topology, information about destination servers, informationabout network link status, and/or the like. To obtain the knowledge ofnetwork topology, the controller 220 may, for example, send discoverypackets throughout the network in a known way, where responses to thediscovery packets may be used by the controller 220 to determine thenetwork topology.

An example of how using the controller 220 to make routing decisions isdescribed with reference to FIGS. 3A-3C. In these figures, thick arrowsrepresent information being sent between devices, with the arrowindicating the direction of information flow. Referring to FIG. 3A, theclient device 201 in this example may send a data packet to the networkdevice 205 (which may be, for example, an edge router of the network200), as indicated by arrow 301. The data packet may be sent directlybetween the client device 201 or via one or more communication linksand/or sub-networks. The data packet may be, for example, an IP packet,an Ethernet packet, a TCP/IP packet, one type of packet encapsulatedinside another type of packet, or any other type of packet. In any ofthese cases, the packet may include information, usually in a header,indicating a source address of the packet (e.g., the address of theclient device 201 and/or of a device sending the packet to the networkdevice 205 on behalf of the client device 201) and a destination addressof the packet. The source and destination addresses may be in the formof any network addresses, such as but not limited to IP addresses. Thepacket (such as in the header) may also identify other information suchas an protocol (e.g., an IP protocol number), one or more ports (such assource and destination port numbers), a quality of service marking suchas a DSCP marking, and/or the like. In response to receiving the packet,the network device 205 may examine the packet to extract information(such as from the header) to allow the network device 205 to determinewhether the packet is part of an existing flow. For instance, thenetwork device 205 may recognize that the packet has the same sourceaddress and destination address as other packets that it has recentlyforwarded, and so it may consider this packet to be part of the sameflow as those other packets. To determine whether the received packet ispart of an existing flow, the network device 205 may compare informationfrom the packet with flow instructions (such as, for example, flow tableentries contained in a flow table) stored by and/or otherwise accessibleto the network device 205. The flow table entries (and/or other flowinstructions) may be updated in accordance with information previouslyprovided by the controller 220. In the present example, it will beassumed that the received packet is not part of an existing flow, asdetermined by the network device 205. In response to such adetermination, the network device 205 may send a request to thecontroller 220, as indicated by arrows 302 and 303.

In response to the request, the controller 220 may determine anappropriate route and/or destination for the packet. As explainedherein, the controller 220 may take one or more factors into account tomake this determination, such as but not limited to current reportedutilizations or other status of one or more of the servers, predictedfuture utilizations or other status of one or more of the servers,congestion, errors, or other problems detected on one or morecommunication paths between the client device 201 and one or more of theservers, an identity of the client device 201 and/or of a user of theclient device 201, a characteristic of the message, the time of day, theday of week, and/or a preference of the user of client device 201 and/orotherwise associated with the client device 201.

Assume, for this example, that the controller 220 determines that theroute for this flow will be from the network device 205 to networkdevice 213, then to the network device 214, and then to Server A. Inmaking the decision, the network device 213 may have selected from apossible pool of servers such as any of Servers A-E. In this case,referring to FIG. 3B, the controller 220 may send instructions to eachof these three network devices for how to route the data, as indicatedby arrows 304, 305, and 306. For example, the instructions to thenetwork device 205 may indicate that it should forward packets of theflow to the network device 213, the instructions to the network device213 may indicate that it should forward packets of the flow to thenetwork device 214, and the instructions to the network device 214 mayindicate that it should forward packets of the flow to Server A. Theinstructions may also include instructions for routing return packets inthe flow from Server A to the client device 201, although the return(downstream) route may or may not be the same as the upstream route fromthe client device 201 to Server A. In addition, the instructions to thenetwork device 214 (the penultimate device providing the last upstreamhop) may include instructions for modifying the destination addresses ofthe upstream packets in the flow to the ultimate destination (e.g.,Server A), and/or instructions for modifying the source address of thedownstream return packets in the flow to the service address the flow isfor (e.g., an address expected by the client device 201 so that loadbalancing between the pool of origin servers may be transparent to theclient device 201).

The network devices 205, 213, and 215 may update their respective flowtables (each of the network devices may maintain their own unique flowtables) in accordance with the instructions and then forward the packet(and future packets in the flow) in accordance with the updated flowtables. Thus, in the present example, the packet may be routed asindicated in FIG. 3C by arrows 307, 308, and 309.

As shown by way of example in FIGS. 4 and 5 , the packet may have beenrouted to other servers and/or using other paths. As previouslydiscussed, the controller 220 may choose not only the route of thepacket but also the destination of the packet. Where the destination ischosen from a load-balancing pool of servers (such as Servers A-E), thecontroller 220 may holistically choose both the route and thedestination together as part of a single decision-making process.Examples of such a process will be described further below withreference to FIGS. 7-9 . Nonetheless, it is pointed out here that thecontroller 220 could have alternately determined that the packet shouldbe routed to, for instance, Server E (as indicated by the arrows in FIG.4 ) or Server C (as indicated by the arrows in FIG. 5 ). In each ofthese cases, the controller 220 may selectively send appropriate routinginstructions to each of the network devices in the respective routepath. For instance, in the example of FIG. 4 , the controller 220 maysend instructions to the network devices 205, 213, 212, and 216, and inthe example of FIG. 5 , the controller 220 may send instructions to thenetwork devices 205, 206, and 208.

FIG. 6 illustrates general hardware and software elements of an examplecomputing device 600 that may be used to implement any of the variouselements discussed herein and/or depicted in any of the figures. Thecomputing device 600 may include, for instance, one or more processors(such as processor 601), which may execute computer-readableinstructions of a computer program to perform, or otherwise cause tooccur, some or all of the features described herein. The instructionsmay be stored in any one or more types of computer-readable media toconfigure the operation of the processor 601. For example, theinstructions may be stored in one or more memory chips (e.g., aread-only memory (ROM) 602 and/or a random access memory (RAM) 603), ahard disk drive, removable storage media 604 (such as a Universal SerialBus (USB) drive, FLASH drive, compact disk (CD) or digital versatiledisk (DVD), or floppy disk drive), and/or any other desired storagemedium. Instructions may also be stored in an attached (or internal)hard drive 605. The computing device 600 may include one or more outputdevices, such as a display 606 (e.g., an external television) ormultiple displays, and may include one or more output device controllers607, such as a video processor. The computing device 600 may alsoinclude or be coupled with one or more user input devices 608, such as aremote control, keyboard, mouse, touch screen, microphone, video camera,etc. The computing device 600 may also include one or more networkinterfaces, such as a network input/output (I/O) circuit 609 (e.g., anetwork card) to communicate with an external device and/or network 610(which may be or otherwise include, for example, network 200). Thenetwork input/output circuit 609 may be a wired interface, wirelessinterface, or a combination of the two. In some embodiments, the networkinput/output circuit 609 may include a modem (e.g., a cable modem), andthe external device/network 610 may include any elements such as any ofthe elements in FIG. 1 or 2 (and/or communication links therewith), thenetwork 200, an in-home network, the Internet, an intranet, a wide-areanetwork, a local-area network, a provider's wireless (e.g., Wi-Fi and/orcellular), coaxial, fiber, or hybrid fiber/coaxial distribution system(e.g., a DOCSIS network), and/or any other desired network.

Modifications may be made to add, remove, combine, divide, etc.components of the computing device 600 as desired. Additionally, thecomponents illustrated may be implemented using basic computing devicesand components, and the same components (e.g., the processor 601, theROM storage 602, the display 606, etc.) may be used to implement any ofthe other computing devices and components described herein. Forexample, the various components herein may be implemented usingcomputing devices having components such as a processor executingcomputer-executable instructions stored on a computer-readable medium.Some or all of the entities described herein may be software based, andmay co-exist in a common physical platform (e.g., a requesting entitycan be a separate software process and program from a dependent entity,both of which may be executed as software on a common computing device).

One or more aspects of the present disclosure may be embodied in theform of, or otherwise utilize, computer-usable data and/or ascomputer-executable instructions, such as in one or more programmodules, executed by one or more computing devices. Generally, programmodules may include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types when executed by a processor in a computer or otherdata processing device. The computer executable instructions may bestored on one or more computer readable media such as a hard disk,optical disk, removable storage media, solid state memory, RAM, etc. Aswill be appreciated by one of skill in the art, the functionality of theprogram modules may be combined or distributed as desired in variousembodiments. In addition, the functionality may be embodied in whole orin part in firmware or hardware equivalents such as integrated circuits,field programmable gate arrays (FPGA), and the like. Particular datastructures may be used to more effectively implement one or more aspectsof the disclosure, and such data structures are contemplated within thescope of computer executable instructions and computer-usable datadescribed herein.

FIGS. 7, 8, and 9 illustrate steps of an example process that may beperformed in accordance with one or more aspects as described herein.Any of the steps may be performed by any element, such as but notlimited to any of the elements described in conjunction with and/ordepicted in FIGS. 1-6 . The process may be used for, e.g., planning theroute of and/or routing packets through a network. By using a processsuch as described herein with reference to FIGS. 7-9 , packet flows maybe distributed to various servers in such a way to implement aload-balancing function. As explained above, such a load-balancingfunction may be integrated with and inter-dependent on the routing ofthe packets. For instance, both the route and destination of a packetmay be based on load-balancing considerations.

Referring to FIG. 7 , at step 701, a new packet may be ready to be sent,such as by the client device 201. The packet, which may be a TCP/IPpacket, for example, may include a header that identifies, among otherinformation, a flow. The flow may include or otherwise be identified by,for instance, an indication of the packet's source address (e.g., sourceIP address) and destination address (e.g., destination IP address), andpossibly one or more other items of information. In some examples, thepacket may be a TCP/IP SYN packet, however the packet may be any otherpacket. The packet may be sent to a network device for further routing.For instance, the packet may be sent to, and received by, network device205, as indicated by way of example in FIG. 3A by the arrow 301. Thus,at step 702, the packet may be received in this example by the networkdevice 205.

As explained above, the network device 205 (and any other networkdevices) may maintain a flow table that associates each of a pluralityof flows (such as OpenFlow flows) or other classifications of datapackets with information that identifies at least a portion of a paththrough a network. At step 703, the receiving device (in this example,the network device 205) may perform a lookup of its own flow table todetermine whether the packet identifies an established flow or a newflow. For example, the flow table of a given network device may includeinformation such as shown below in Table 1. The flow table for eachnetwork device may be stored locally in a computer-readable medium, suchas memory, of the respective network device. Alternatively, two or moreof the network devices may share a common computer-readable medium tostore their flow tables.

TABLE 1 Example Flow Table FLOW NEXT HOP flow 1 network device 213 flow2 network device 206 flow 3 network device 213

In the example of Table 1, three flows (generically referred to hereinas flow 1, flow 2, and flow 3) are each associated with the next hop ina route for the flow. For example, the flow table here indicates thatpackets in flow 1 should be forwarded to the network device 213, andpackets in flow 2 should be forwarded to the network device 206. Theinformation in the example flow table is shown generically; the actualinformation contained in the flow table may include, for instance, whichport to send a packet through, protocol settings for sending the packet,an address of the next hop, and/or the like. Moreover, while flows suchas “flow 1” are generically indicated in Table 1, the flows may beindicated in other ways, such as a destination address and destinationport pair. For instance, in some embodiments, each pair of destinationaddress/port may define a flow. In addition or alternatively, each flowmay be associated in the flow table with one or more executableinstructions, such as OpenFlow instructions, for how to route and/orotherwise handle packets for the flow and/or to alter the packets suchas by changing address information and/or other information in thepacket headers. Flows that are not established may either be entirelymissing from the flow table or associated with a state other than anestablished state. As will be described further below, the flow table toa given network device may be populated in accordance with informationprovided by the controller 220. To determine the flow of a packet, oneor more elements of information included in the packet header may beused, such as source address, destination address, protocol number,source port, and/or destination port. If the flow table indicates thatthe flow of the packet is an existing (e.g., established) flow, then theprocess may proceed to step 718 to route in accordance with the normalroute already established for the flow. Otherwise, the process mayproceed to step 704 to determine a route.

Assuming that the flow is determined at step 703 to be new, then at step704, the network device 205 may contact the controller 220 in an attemptto obtain instructions as to how to route the packet. As explainedabove, some or all of the network devices in a network (including, inthe present example, the network device 205) may be configured toutilize OpenFlow and/or another communications protocol to communicatewith the controller 220. In the present example, at step 704, thenetwork device 205 may send a request (such as an OpenFlow request) tothe controller 220, where the request may identify information about thepacket such as the flow of the packet. For instance, the request mayinclude some or all of the header of the packet, and/or a hash key ofsome or all of the information in the header of the packet. In someexamples, the request may include some or all of the followinginformation from the header of the packet: the source address, thedestination address, the IP protocol number, the source port, and/or thedestination port. In other examples, the network device 205 may generatea hash key (also referred to herein as a flow) of some or all of theseand/or other items of information and the request may include the hashkey. In still further examples, the controller 220 and/or another deviceother than the network device 205 may generate the hash key. Forinstance, where the request includes some or all of the headerinformation elements listed above, the controller 220 may generate thehash key from those information elements as received in the request.

FIG. 3A shows an example of the routing of the request from networkdevice 205 to the controller 220. In the present example, the requestmay be routed from the network device 205 to the network device 213 (asindicated by the arrow 302), which may then route the request to thecontroller 220 (as indicated by arrow 303). To be appropriately routedto the controller 220, the request may itself include at least onepacket that identifies the controller 220 as a destination address.Requests to the controller 220, and any responses from the controller220, may be in packets with specially designated flows associated withrouting instructions in the flow tables of the various devices of thenetwork.

The controller 220 may receive the request, and in response, at step706, the controller 220 may determine a flow (e.g., hash key) of thepacket based on the request. To determine the flow, and if not alreadydetermined by another device such as the network device 205, thecontroller 220 may hash together some or all of the information providedin the request. For example, the controller 220 may perform a five-tuplehash of the source IP address, the destination IP address, the IPprotocol number, the source port, and the destination port. In otherexamples, more generally an N-tuple hash may be performed, where N isany whole number greater than one. As explained above, such hashing mayalternatively be performed by another device, such as by the networkdevice 205.

At step 707, the controller 220 may determine, based on the flow (e.g.,the hash key), whether the packet is addressed to a virtual IP address(VIP) or other virtual address. A virtual address is an address that isnot necessarily associated with only a single physical destinationand/or device, but rather one that may be associated with a plurality ofdifferent physical destinations/devices. For example, referring to FIG.3A, a single virtual address may be associated with a pool of physicaldestination devices, such as some or all of Server A, Server B, ServerC, Server D, and Server E. Virtual addresses may be useful toimplementing load balancing services, where packets identifying thevirtual address may be distributed to one or another of a plurality ofservers based on a load balancing policy. For example, if the user ofthe client device 201 were to browse a particular website, the websiteaddress entered into the browser window may be translated by the domainname system to an IP address that is a virtual address associated with aplurality of different redundant web servers. The user of the clientdevice 201 may not care whether (or even be aware of) the packet is sentto one of the redundant servers or another of the redundant servers, asthe returned web page information may look identical to the user. Fromthe user's perspective, the web page may be more responsive than if onlya single web server were used for all web page requests. While in theabove example the multiple servers may provide a set of redundant andidentical services, in other examples the multiple servers associatedwith a virtual address may not provide identical information and/orservices to the user of the client device 201.

To determine whether the packet is addressed to a virtual address atstep 707, the controller 220 may maintain a list or other informationidentifying flows that are associated with a virtual address. Thecontroller 220 may obtain such information from any source, such as fromthe servers (e.g., any of Servers A-E), another device, via manual dataentry, etc. In some examples, any servers that are part of VIP or othervirtual address group may identify themselves to the controller 220,possibly along with any other information such as status informationabout the server (e.g., server utilization, data connectivityexperienced by the server, server capacity, etc.) The identification ofwhich servers belongs to which virtual addresses may be performedindependently by the servers and/or in response to a polling requestfrom the controller 220, and may be performed at any time such as atboot-up or resetting of the server, at boot-up or resetting of thecontroller 220, periodically, etc. If the controller 220 determines thatthe flow is associated with a virtual address, then the process mayproceed to step 711. Otherwise, the process may proceed to step 708, inwhich a normal routing policy is implemented to generate normal flowtable entries at step 709 for routing packets of the identified flow tothe destination and/or for returning packets from the destination.

Assuming that the flow of the packet is determined to be associated witha destination address that is a virtual address, then at step 711, thecontroller 220 may use a predetermined routing policy to determine, atstep 712, a routing path for the packet. The routing policy used by thecontroller 220 may be any routing policy that is based on any number ofpossible inputs. For example, the routing policy may be or otherwiseinclude a least-cost routing policy, a shortest path routing policy(e.g., an open shortest path first policy), and/or a least-latencyrouting policy, preferred paths. In some embodiments, the routing policymay include a combination of two or more different routing policies(sub-policies), where results of each of the sub-policies may becombined to determine a single desired route. Also, in some embodiments,the routing policy may utilize inputs other than the flow (e.g., thehash key) and/or inputs other than information provided from the packet.For example, the routing policy may utilize inputs such as measurednetwork congestion at different portions of a network and/or ondifferent paths within the network, time of day, day of the week,current load on one or more of the servers associated with the virtualaddress, expected (predicted) load and/or congestion of the serversand/or the network, and/or other information. At step 712, the path maybe determined not just for the direction the packet is flowing in (e.g.,in the upstream direction from the client device 201 toward one ofServers A-E), but also for the return (e.g., downstream) path ifdesired. The upstream and downstream paths may or may not be the samepaths (e.g., may or may not utilize the same network devices). Moreover,while the initial packet in a flow has been characterized as beingupstream from a client device to a server, the initial packet in a flowthat triggers the “new” branch from step 703 may alternatively be adownstream packet from a server to a client device.

In determining the route, the routing policy may also be used todetermine the ultimate destination of a packet that is addressed to avirtual address. For example, if Server A and Server B are bothassociated with the same virtual address to which the packet isdirected, the controller 220 may use known information about networkpath congestion and/or information about (e.g., utilization of, numberof flows being served by, capacity of, etc.) Servers A and/or B todetermine not only the route of the packet, but also whether the packetshould be routed to Server A or Server B. Thus, the particular physicaldestination (e.g., Server A versus Server B) may be determined as partof the routing determination, in some cases before the packet is evenrouted through the network. More generally, determining the route of thepacket at step at steps 711 and 712 may involve selecting both aphysical destination (e.g., a real IP address or other real networkaddress) from a plurality of candidate physical destinations associatedwith a virtual address as well as determining a route to the selectedphysical destination. Moreover, the selection of the destination maydepend upon information about network conditions that are independent ofthe destination, such as network path congestion. For example, referringto FIG. 2 , if Servers A and B are both associated with the same virtualaddress to which the packet is addressed, Server B might be selected ifthe controller 220 determines that one or more paths between the networkdevice 205 and the network device 214 are congested, broken, or havesome other problem. Likewise, Server A might be selected if thecontroller 220 determines that one or more paths between the networkdevice 205 and the network device 208 are congested, broken, or havesome other problem. In some examples, multiple factors may be combinedtogether to determine the ultimate physical destination and route. Forinstance, the controller 220 may take into account both networkcongestion and the utilization of Servers A and B when determining theroute and the physical destination, such as by reducing these factors tomultiple values that may be combined together (e.g., as a weightedaverage or using a more complex algorithm depending upon the routingpolicy) to determine weights of candidate route/destination pairs. Theroute/destination pair having a weight indicating the best choiceaccording to the routing policy might then be selected by the controller220.

In yet another example, FIG. 9 illustrates a flow chart describing anexample of the controller 220 determining a route and destination inaccordance with steps 711 and/or 712. At step 901, an indication of thevirtual address may be received, such as provided from the flowinformation at step 706 and/or step 707. The indication of the virtualaddress may include, for instance, the value of the virtual addressand/or a listing of the physical/actual addresses associated with thevirtual address. At step 902, the controller 220 may obtain one or morerouting decision inputs for use in implementing the routing policy. Suchrouting decision inputs, if any may include those discussed above, suchas network congestion and/or other problems, information about candidatedestination servers (e.g., destination server utilization, capacity,number and/or types of flows currently being served, etc.) a clockinput, network latency measurements, network topology, and/or the like.These routing decision inputs may be polled and/or may have already beenreceived and stored by the controller 220 prior to step 902. At step905, the controller 220 may determine one or more candidate physicaldestinations, out of the set of physical destinations associated withthe virtual address, in accordance with the routing policy. A candidatedestination may include, for example, a server (e.g., one or more ofServers A-E), a cluster of multiple servers, a gateway/edge deviceproviding connectivity to a separate network that may or may not use therouting methodology described herein (e.g., that may or may notcommunicate with the controller 220 or a similar controller), or anothertype of network device. This decision may be at least partially based onthe routing decision inputs obtained at step 902. For example, if thevirtual address is associated with Server A, Server B, Server C, andServer D, and if Server C is currently over-utilized (e.g., utilizedmore than a predetermined threshold amount of utilization), then therouting policy may dictate that Server C is not presently a candidatefor selection as a physical destination. In this example, the controller220 might therefore determine that the candidates for a physicaldestination are Server A, Server B, and Server D.

At step 906, the controller 220 may then run a plurality of routingcomputations for the various multiple physical destination candidates.For example, if there are N physical destination candidates (where N isany whole number), then at least N independent routing computations maybe run to determine at least N routes, at least one route (e.g., flowpath) for each of the physical destination candidates. Some or all ofthe routes may each be multi-hop routes; that is, some or all of theroutes may each contain a plurality of hops between a plurality ofnetwork devices. In some examples, more than one possible route may becomputed for a given physical destination candidate. The N or morerouting computations may be run in parallel (e.g., simultaneously ornearly simultaneously) or one after the other. Moreover, each routingcomputation may also be computed in accordance with the routing policy.For example, each routing computation may use the network topology knownto the controller 220 and may be computed based on a shortest-routealgorithm, a least-cost algorithm, or a least-latency algorithm. Inother embodiments, more than one algorithm (e.g., shorted-route andleast-cost) may be used to compute a route. In still furtherembodiments, two or more routes may be computed for each physicaldestination, where the various routes for a given physical destinationmay be computed using different algorithms (e.g., one route for thephysical destination computed using shortest-route and another route forthe same physical destination computed using least-cost).

At step 907, the controller 220 may use the status information describedabove to discard any of the routes computed at step 906 that would beinoperable or otherwise undesirable. For example, the controller 220 maydetermine, using the status information, for each of the computedroutes, whether the computed route has a problematic link or path (e.g.,a broken link), an overly slow and/or congested path (e.g., the latencyand/or congestion of the route or portion thereof exceeds apredetermined threshold), a destination server for the route is overlyutilized (e.g., utilization is more than a predetermined thresholdvalue), not operational, not accepting new flows, etc.

At step 908, the best route/physical destination pair, from theremaining computed routes that were not discarded, may then be selectedby the controller 220, again in accordance with the routing policy. Forinstance, the controller 220 may analyze one or more characteristics ofthe various computed route/destination pairs and then select the bestone per the routing policy. As an example, where the routing policy isto ultimately select the shortest route, the controller 220 may selectthe shortest route from all of the N or more computed routes. As anotherexample, the controller 220 may assign a score to each of the determinedroutes, where the scoring algorithm may depend upon the routing policy.The route with the best (e.g., highest) score may then be chosen at step908. In further embodiments, steps 907 and 908 may be integrated in thatimplementing the routing policy may cause one or more routes to bediscarded such as on the same basis as they would have been discarded instep 907. Returning to FIG. 7 , the route/destination pair selected atstep 908 may be output as a result of performing step 712.

The above are examples of how load-balancing in accordance with theconcepts described herein may allow the controller 220 to make aholistic decision in which route and destination selection may beintertwined and interdependent. Moreover the load-balancing may be notjust a balancing amongst multiple destination servers, but also abalancing amongst multiple possible routes through the network.

Next, at step 713, the controller 220 may determine whether the any ofthe network devices will need to be instructed to rewrite the header ofthe data packet so as to modify the source address and/or thedestination address indicated by the header. For example, as explainedabove, for an upstream path (e.g., from the client device 201 to ServerA), it may be desirable for the penultimate device (the device justprior to the ultimate destination, which is Server A in this example) torewrite the destination address in the header of the data packet to bethe address of the ultimate destination. Likewise, for a downstream path(e.g., from Server A to the client device 201), it may be desirable forthe device immediately downstream from Server A (the origin server inthis example) to rewrite the source address in the header of the datapacket to be the address of Server A. If the determination at step 713is that the header does not need to be rewritten, then the process maymove to step 714.

If the determination at step 713 is that the header is to be rewritten,then the process may move to step 801 (FIG. 8 ). At step 801, thecontroller 220 may determine whether a header rewrite is also needed fora return path for the flow. For example, a return path may not exist ifthe flow is a unidirectional flow. In other words, some flows may haveonly one path (e.g., upstream or downstream path) while other flows mayhave both an upstream path and a downstream path, where one of theupstream or downstream path is a return path. If a header rewrite is notneeded for a return path for the flow, then the process may move to step804. Otherwise, the process may move to step 803.

At step 803, the controller 220 may identify, based on the pre-computedreturn path of the flow that was determined at step 712, which of thenetwork devices in the return path receives the first hop after thesource of the return path. For example, if the return path is fromServer A to the client device 201, then in the example network of FIG. 2, the network device receiving the first hop would be the network device214. The controller 220 may prepare instructions for the identifiednetwork device to modify the source address in the headers of returnpackets of the flow. For example, the source address may be modified tomatch the virtual destination address that was originally provided bythe client device 201, or another source address that may be expected bythe client device 201. Modifying the source address may make the loadbalancing transparent to the client device 201, in that the actualorigin server address may not be included in the packet header when thepacket arrives at the client device 201. In other embodiments, thecontroller 220 may select a different network device in the path tomodify the destination address.

At step 804, the controller 220 may determine whether to rewrite thereturn path of the headers of the packets of the flow. For instance, thecontroller 220 may identify, based on the pre-computed original(non-return) path of the flow that was determined at step 712, which ofthe network devices in the path sends the last hop before the ultimatedestination of the path. For example, if the path is from the clientdevice 201 to Server A, then in the example network of FIG. 2 , thenetwork device sending the last hop would be the network device 214. Thecontroller 220 may prepare rewrite instructions for the identifiednetwork device to rewrite the destination address in the headers ofpackets of the flow. For example, the destination address may berewritten to match the actual address of the ultimate destinationselected at step 712 (e.g., the address of Server A). In otherembodiments, the controller 220 may select a different network device inthe return path to rewrite the source address.

At step 805, the controller 220 may rewrite the original path and/or thereturn path instructions to incorporate the header address rewriteinstructions. Then, the process may proceed to step 714 (FIG. 7 ).

Referring again to FIG. 7 , at step 714, the controller 220 may generatean actual set of flow table entries containing instructions (e.g.,rules) for routing the packet from, in this example, the network device205 to the selected physical destination using the computed pathresulting from step 712. In addition, this may involve determining whichnetwork devices are part of the flow path. For example, assume thatServer A is selected as the physical destination and the flow path isthe path from the network device 205 to network device 213, then to thenetwork device 214, and then to Server A. This flow path is indicated inFIG. 3C by way of the arrows 307, 308, and 309. In such an example, thecontroller 220 may determine that the network devices 205, 213, and 214are part of the flow path and generate appropriate flow table entriesfor one or more (e.g., all) of the network devices in the determinedflow path. Thus, referring to the present example, flow table entriesmay be generated for some or all of the network devices 205, 213, and/or214. The flow table entries for the network device 205 may indicate thatpackets for the flow should be forwarded to the next hop of the networkdevice 213, the flow table entries for the network device 213 mayindicate that packets for the flow should be forwarded to the next hopof the network device 214, and the flow table entries for the networkdevice 214 may indicate that packets for the flow should be forwarded tothe next hop of Server A. The flow table entries may also includeinstructions to change addresses and/or other information in the packetheader as determined from step 713. In some embodiments, the flow tableentries are only generated for and/or sent to the network device thatmade the request to the controller 220. In other embodiments, the flowtable entries are generated for and/or sent to each of (or a subset of)the network devices in the computed path for the flow.

At step 717, the controller 220 may determine whether there are anynetwork devices to program with flow table entries that were generatedat step 709 or at step 714. If the answer is “no,” then the process mayproceed to step 718, at which point, the controller 220 might reply tothe original request by instructing the network device 205 to route thepacket normally. However, in the present example, the answer would be“yes,” since the network devices 205, 213, and 214 would need flow tableentries containing instructions for handling the packet. Accordingly,the process may proceed to step 719.

At step 719, the controller 220 may respond to the original request bysending one or more sets of flow table entries back to the determinednetwork devices. The loop of steps 717 and 719 may repeat until alldesired network devices have been updated with appropriate flow tableentries. The updating of the network devices 205, 213, and 214 (forexample) is shown by way of the arrows 304, 305, and 306 in FIG. 3B. Inother embodiments, the controller 220 may send flow table entries and/orother information (e.g., other instructions) only to the requestingnetwork device (in this example, the network device 205). Then, as thepacket flows through the path, each subsequent network device in thepath may repeat the request/response protocol to determine where to sendthe packet next. For example, after the network device 205 sends arequest to and receives a response from the controller 220 with flowtable entries or other instructions to forward the packet to the networkdevice 213, then the network device 213 may similarly send a request toand receive a response with flow table entries and/or other instructionsfrom the controller 220 to forward the packet to the network device 214,and so on.

Once the network device 205 and any other network devices receive theflow table entries and/or other instructions from the controller 220,the network devices may update their respective flow tables and/or otherworking instruction sets with the flow table entries and/or otherinstructions. For example, if the above-described packet received at thenetwork device 205 is for flow 4, then the flow table for the networkdevice 205 may be updated to include flow 4 associated with flow tableentries for, e.g., sending flow 4 packets to the next hop—the networkdevice 213. Then, the network device 205 and the remaining networkdevices in the flow path may forward the packet as instructed by theirrespective flow tables (which may also have been updated withinformation received from the controller 220), such as shown by the flowpath indicated by the arrows 307, 308, 309 of FIG. 3C.

While Server A is used as the physical destination in an above-describedexample, another server in the pool of servers associated with thevirtual address may have been selected at steps 711 and 712 (in FIG. 7 )and/or step 907 (in FIG. 9 ). For instance, FIG. 4 shows an example flowpath in which Server E is selected as the physical destination, and theassociated flow path would be as illustrated by the thick arrows in FIG.4 . As another example, FIG. 5 shows a flow path associated withselecting Server C as the physical destination, where the flow path inthis example is illustrated in FIG. 5 using thick arrows. In each ofthese examples, any of the network devices that receives the packetwithout an existing flow may perform the process of FIGS. 7-9 startingwith the “new” branch of step 703. While in each of these examples thefirst network device to receive the packet without an existing flowwould be the network device 205, in other examples it may be a differentnetwork device.

In fact, even with the examples of FIGS. 3-5 , a different networkdevice might also receive the packet after being partially routedthrough a flow path and yet consider the packet not to be associatedwith an existing flow (according to the network device's flow table).This may occur if, for example, that network device's flow table has notbeen properly updated. For instance, in FIG. 4 , if the flow table ofthe network device 212 does not list the flow of the packet as anexisting flow, then the network device 212 may also use the process ofFIGS. 7-9 to send a request to the controller 220, and in response thecontroller 220 may send instructions that may be used to update the flowtable to as to appropriately route the packet to the next hop.

Other situations exist where a packet that has been partially routed maystill be considered associated with a non-existing flow during routing.For instance, the controller 220 may continuously monitor the conditionsthat impact existing flows in the network 200, and reconfigure the pathsfor the flows as conditions change. By way of example, if one or morenetwork devices report excessive congestion (e.g., above a predeterminedthreshold amount of congestion) or other problem in the pre-computedpath of a flow, the controller 220 may, in response, dynamically modifythe flow path of packets in the flow. To do this, the controller 220 canwithdraw (erase, delete, invalidate, etc.) the entry for that flow inthe flow table(s) for one or more of the network devices in the flowpath. For example, assume that during routing of a packet through a flowpath as in FIG. 4 , the link between the network device 213 and thenetwork device 212 goes down (e.g., is disconnected) or experiencesexcessive congestion. In response to detecting this outage or otherproblem with the link, the controller 220 may send out a message to thenetwork device 213 (and possibly also the network devices 212) towithdraw one or more flow table entries for one or more flows that aresupposed to travel through that link. When a packet for one of thosewithdrawn flows reaches the network device 213 (as passed from thenetwork device 205), then the network device 213 may determine, at step703, that the flow for the packet is a new flow, and so in response thenetwork device 213 may send a request (per step 704) to the controller220 for instructions as to how to handle packets for the flow. Inaddition to re-routing in response to experienced network conditions,the controller 220 may preemptively move flows to different paths inresponse to scheduled or predicted network or other infrastructureevents. Other factors that may trigger a re-routing of a flow mayinclude, for instance, monitored present or predicted availability ofrenewable energy sources coupled to and/or powering network devices ofthe network 200, the past performance of certain links, paths, and/ornetwork devices in the network 200, the current cost of using certainpaths and/or network devices or servers in the network 200, and/or thelike.

As yet another example, there may be one or more network devices that donot participate in the flow methodology described herein (e.g., the oneor more network devices may not communicate with the controller 220).This may be due to, for example, the one or more network devices notbeing configured to do so, or not having appropriate connectivity to thecontroller 220, or for another reason. Regardless of the reason, the oneor more network devices might use traditional routing/switching methods(e.g., self-determine a next hop for a new flow without consulting thecontroller 220). Thus, a packet may be partially routed using a routedictated by the controller 220 until the packet reaches the one or morenetwork devices mentioned above, in which case the packet may be routedoutside of the route dictated by the controller 220. In this case, thenext network device that uses the methodology described herein mayconsider the packet part of a new flow and in response requestinstructions for how to route packets in the flow from the controller220. In response, the controller 220 may recalculate the route in wholeor in part (e.g., the remainder of the route) based on which networkdevice is making the request for instructions, and update any networkdevices with new/revised routing instructions as appropriate.

In still further embodiments, the controller 220 may aggregate flowrules and program flow actions to apply to a wider set of clienttraffic. This may have the effect of reducing the amount of futurerequests that are sent to the controller 220 from the same clientdevice, the same subnet, and/or the same geographic region.

In the example embodiments described above, the various features andsteps may be combined, divided, omitted, rearranged, revised and/oraugmented in any desired manner, depending on the specific outcomeand/or application. Various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements as are made obvious by this disclosureare intended to be part of this description though not expressly statedherein, and are intended to be within the spirit and scope of thedisclosure. Accordingly, the foregoing description is by way of exampleonly, and not limiting.

1. A system comprising: a computing device; and a first network device;wherein the computing device comprises: one or more first processors;and memory storing first instructions that, when executed by the one ormore first processors of the computing device, configure the computingdevice to: receive a request message; generate, based on the requestmessage, a flow identifier; determine a route associated with the flowidentifier, wherein the route comprises: the first network device, of aplurality of network devices, a second network device, of the pluralityof network devices, and an intermediary network device, of the pluralityof network devices, between the first network device and the secondnetwork device; and send, to the first network device and theintermediary network device, the flow identifier and an instruction toroute data packets based on the route; and wherein the first networkdevice comprises: one or more second processors; and memory storingsecond instructions that, when executed by the one or more secondprocessors of the first network device, configure the first networkdevice to: receive the flow identifier and the instruction to route datapackets.
 2. The system of claim 1, wherein the flow identifier is basedon one or more header values of a first data packet.
 3. The system ofclaim 1, wherein the flow identifier is based on one or more headervalues of a first data packet, and the first instructions, when executedby the one or more first processors, configure the computing device tocause the first network device to forward, based on the flow identifierand the instruction to route data packets, the first data packet.
 4. Thesystem of claim 1, wherein: the request message comprises a destinationaddress of a first data packet and the flow identifier is based on thedestination address; the destination address is a virtual destinationaddress corresponding to a plurality of servers; and each of theplurality of servers stores content requested by the first data packet.5. The system of claim 1, wherein the route comprises one or more of aplurality of servers storing content.
 6. The system of claim 1, whereinthe first instructions, when executed by the one or more firstprocessors, configure the computing device to determine the route byselecting, based on a destination address of a first data packet, theroute from a plurality of candidate routes.
 7. A system comprising: acomputing device; and a client device; wherein the computing devicecomprises: one or more first processors; and memory storing firstinstructions that, when executed by the one or more first processors ofthe computing device, configure the computing device to: receive, fromthe client device, a request for content; generate, based on therequest, a flow identifier; determine a forward route associated withthe flow identifier, wherein the forward route comprises: the clientdevice, a destination device storing the content, a first networkdevice, of a plurality of network devices, between the client device andthe destination device, and an intermediary network device, of theplurality of network devices, between the client device and thedestination device; and send, to the first network device and theintermediary network device, the flow identifier and an instruction toroute data packets, from the client device to the destination device,based on the forward route; and wherein the client device comprises: oneor more second processors; and memory storing second instructions that,when executed by the one or more second processors of the client device,configure the client device to: send the request for content.
 8. Thesystem of claim 7, wherein the flow identifier is based on one or moreheader values of a first data packet.
 9. The system of claim 7, wherein:the request comprises a destination address of a first data packet andthe flow identifier is based on the destination address; the destinationaddress is a virtual destination address corresponding to a plurality ofservers; and each of the plurality of servers stores content associatedwith the request for content.
 10. The system of claim 7, wherein thefirst instructions, when executed by the one or more first processors,configure the computing device to determine the forward route byselecting the forward route based on: a destination address of thedestination device, the plurality of network devices, and one or more ofa least-cost algorithm or a shortest-path routing policy.
 11. The systemof claim 7, wherein the first instructions, when executed by the one ormore first processors, configure the computing device to: based on thedestination device, modify a destination address, in one or more headervalues of the data packets, to be a final network address.
 12. Thesystem of claim 7, wherein the first instructions, when executed by theone or more first processors, configure the computing device todetermine the forward route by determining the forward route based on adetermination that the request for content is not associated with anexisting flow identifier.
 13. The system of claim 7, wherein the firstinstructions, when executed by the one or more first processors,configure the computing device to: determine a return route associatedwith the flow identifier, wherein the return route comprises: thedestination device, a second network device, of the plurality of networkdevices, between the client device and the destination device, a secondintermediary network device, of the plurality of network devices,between the client device and the destination device, and the clientdevice; and send, to the second network device and the secondintermediary network device, the flow identifier and a furtherinstruction to route data packets based on the return route.
 14. Acomputing device comprising: one or more processors; and memory storinginstructions that, when executed by the one or more processors,configure the computing device to: receive a request message; generate,based on the request message, a flow identifier; determine a routeassociated with the flow identifier, wherein the route comprises: afirst network device, of a plurality of network devices, a secondnetwork device, of the plurality of network devices, and an intermediarynetwork device, of the plurality of network devices, between the firstnetwork device and the second network device; and send, to the firstnetwork device and the intermediary network device, the flow identifierand an instruction to route data packets based on the route.
 15. Thecomputing device of claim 14, wherein the flow identifier is based onone or more header values of a first data packet.
 16. The computingdevice of claim 14, wherein the flow identifier is based on one or moreheader values of a first data packet, and the instructions, whenexecuted by the one or more processors, configure the computing deviceto cause the first network device to forward, based on the flowidentifier and the instruction to route data packets, the first datapacket.
 17. The computing device of claim 14, wherein: the requestmessage comprises a destination address of a first data packet and theflow identifier is based on the destination address; the destinationaddress is a virtual destination address corresponding to a plurality ofservers; and each of the plurality of servers stores content requestedby the first data packet.
 18. The computing device of claim 14, whereinthe route comprises one or more of a plurality of servers storingcontent.
 19. The computing device of claim 14, wherein the instructions,when executed by the one or more processors, configure the computingdevice to determine the route by selecting, based on a destinationaddress of a first data packet, the route from a plurality of candidateroutes.
 20. A computing device comprising: one or more processors; andmemory storing instructions that, when executed by the one or moreprocessors, configure the computing device to: receive, from a clientdevice, a request for content; generate, based on the request, a flowidentifier; determine a forward route associated with the flowidentifier, wherein the forward route comprises: the client device, adestination device storing the content, a first network device, of aplurality of network devices, between the client device and thedestination device, and an intermediary network device, of the pluralityof network devices, between the client device and the destinationdevice; and send, to the first network device and the intermediarynetwork device, the flow identifier and an instruction to route datapackets, from the client device to the destination device, based on theforward route.
 21. The computing device of claim 20, wherein the flowidentifier is based on one or more header values of a first data packet.22. The computing device of claim 20, wherein: the request comprises adestination address of a first data packet and the flow identifier isbased on the destination address; the destination address is a virtualdestination address corresponding to a plurality of servers; and each ofthe plurality of servers stores content associated with the request forcontent.
 23. The computing device of claim 20, wherein the instructions,when executed by the one or more processors, configure the computingdevice to determine the forward route by selecting the forward routebased on: a destination address of the destination device, the pluralityof network devices, and one or more of a least-cost algorithm or ashortest-path routing policy.
 24. The computing device of claim 20,wherein the instructions, when executed by the one or more processors,configure the computing device to: based on the destination device,modify a destination address, in one or more header values of the datapackets, to be a final network address.
 25. The computing device ofclaim 20, wherein the instructions, when executed by the one or moreprocessors, configure the computing device to determine the forwardroute by determining the forward route based on a determination that therequest for content is not associated with an existing flow identifier.26. The computing device of claim 20, wherein the instructions, whenexecuted by the one or more processors, configure the computing deviceto: determine a return route associated with the flow identifier,wherein the return route comprises: the destination device, a secondnetwork device, of the plurality of network devices, between the clientdevice and the destination device, a second intermediary network device,of the plurality of network devices, between the client device and thedestination device, and the client device; and send, to the secondnetwork device and the second intermediary network device, the flowidentifier and a further instruction to route data packets based on thereturn route.
 27. One or more non-transitory computer-readable mediastoring instructions that, when executed, cause: receiving, by acomputing device, a request message; generating, based on the requestmessage, a flow identifier; determining a route associated with the flowidentifier, wherein the route comprises: a first network device, of aplurality of network devices, a second network device, of the pluralityof network devices, and an intermediary network device, of the pluralityof network devices, between the first network device and the secondnetwork device; and sending, to the first network device and theintermediary network device, the flow identifier and an instruction toroute data packets based on the route.
 28. The one or morenon-transitory computer-readable media of claim 27, wherein the flowidentifier is based on one or more header values of a first data packet.29. The one or more non-transitory computer-readable media of claim 27,wherein the flow identifier is based on one or more header values of afirst data packet, and the instructions, when executed, further cause:causing the first network device to forward, based on the flowidentifier and the instruction to route data packets, the first datapacket.
 30. The one or more non-transitory computer-readable media ofclaim 27, wherein: the request message comprises a destination addressof a first data packet and the flow identifier is based on thedestination address; the destination address is a virtual destinationaddress corresponding to a plurality of servers; and each of theplurality of servers stores content requested by the first data packet.31. The one or more non-transitory computer-readable media of claim 27,wherein the route comprises one or more of a plurality of serversstoring content.
 32. The one or more non-transitory computer-readablemedia of claim 27, wherein the instructions, when executed, cause thedetermining the route by selecting, based on a destination address of afirst data packet, the route from a plurality of candidate routes. 33.One or more non-transitory computer-readable media storing instructionsthat, when executed, cause: receiving, by a computing device and from aclient device, a request for content; generating, based on the request,a flow identifier; determining a forward route associated with the flowidentifier, wherein the forward route comprises: the client device, adestination device storing the content, a first network device, of aplurality of network devices, between the client device and thedestination device, and an intermediary network device, of the pluralityof network devices, between the client device and the destinationdevice; and sending, to the first network device and the intermediarynetwork device, the flow identifier and an instruction to route datapackets, from the client device to the destination device, based on theforward route.
 34. The one or more non-transitory computer-readablemedia of claim 33, wherein the flow identifier is based on one or moreheader values of a first data packet.
 35. The one or more non-transitorycomputer-readable media of claim 33, wherein: the request comprises adestination address of a first data packet and the flow identifier isbased on the destination address; the destination address is a virtualdestination address corresponding to a plurality of servers; and each ofthe plurality of servers stores content associated with the request forcontent.
 36. The one or more non-transitory computer-readable media ofclaim 33, wherein the instructions, when executed, cause the determiningthe forward route further by selecting the forward route based on: adestination address of the destination device, the plurality of networkdevices, and one or more of a least-cost algorithm or a shortest-pathrouting policy.
 37. The one or more non-transitory computer-readablemedia of claim 33, wherein the instructions, when executed, furthercause: based on the destination device, modifying a destination address,in one or more header values of the data packets, to be a final networkaddress.
 38. The one or more non-transitory computer-readable media ofclaim 33, wherein the instructions, when executed, cause the determiningthe forward route based on a determination that the request for contentis not associated with an existing flow identifier.
 39. The one or morenon-transitory computer-readable media of claim 33, wherein theinstructions, when executed, further cause: determining a return routeassociated with the flow identifier, wherein the return route comprises:the destination device, a second network device, of the plurality ofnetwork devices, between the client device and the destination device, asecond intermediary network device, of the plurality of network devices,between the client device and the destination device, and the clientdevice; and sending, to the second network device and the secondintermediary network device, the flow identifier and a furtherinstruction to route data packets based on the return route.